Member of Engineering (Pre-training / Data Acquisition)

📋 Description Design, build, and operate a large-scale web crawler for open data Develop deep crawlers to high-value sources to improve recall Own a long-term data acquisition roadmap with researchers Build observability, monitoring, and debugging tooling for crawl infrastructure Collaborate with pre-training, post-training, and evaluations teams on data priorities Build high-throughput ingestion pipelines for partner data 🎯 Requirements Strong distributed systems background with large-scale data pipelines and crawlers Proficiency in Python; optimize performance and debug production systems Hands-on web crawling or data extraction; HTTP, queues, and parsing at scale Familiarity with AWS and Kubernetes/Docker for high-throughput workloads Awareness of data privacy and responsible crawling practices Nice to have: pre-training LLMs; trillion-scale datasets; translating research to production 🎁 Benefits Fully remote work and flexible hours 37 days/year vacation and holidays 16 weeks of flexible, full-pay parental leave Health insurance allowance for you and dependents Company-provided equipment Well-being, learning, and home office allowances